Skip to content

Boot-window metrics — record turns/cache-creation instead of gating on magic thresholds, surface per-PR delta#470

Merged
clkao merged 8 commits into
mainfrom
spacedock-ensign/boot-metrics-record-and-pr-delta
Jul 3, 2026
Merged

Boot-window metrics — record turns/cache-creation instead of gating on magic thresholds, surface per-PR delta#470
clkao merged 8 commits into
mainfrom
spacedock-ensign/boot-metrics-record-and-pr-delta

Conversation

@clkao

@clkao clkao commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Replaces two frozen, meaningless CI thresholds with real per-PR boot-cost tracking and historical release visibility.

What changed

  • Replace AC-6's hardcoded 60k/60k thresholds with recorded turns/token telemetry
  • Aggregate the release ledger from every PR+main run since the last tag
  • Post a per-PR sticky comment diffing against the latest published release observation
  • Add a backup-then-structural-diff safeguard before any historical release-asset rewrite
  • Cover release.yml's per-run-subdirectory download script with a direct positive-path test

Evidence

  • Full Go suite: 15/15 packages passed (build, vet, test) across two validation cycles
  • Detached adversarial audit on a throwaway checkout confirmed the fix is load-bearing after one REJECT→fix→PASS cycle

e7f

Spike Test added 4 commits July 2, 2026 20:33
…nance

Removes the two absolute 60k threshold branches from
assertShallowBootMeasuredTurns (calibrated once against a since-superseded
baseline) and records the greet turn's full token usage as a distinct
shallow-boot-window journeymetrics observation instead, so boot cost rides a
trend instead of a magic-number gate.

Adds RunID/RunURL/CapturedAt provenance fields to journeymetrics.Record,
stamped at EmitRecord time, and fixes release.yml's journey-ledger job to
download each discovered Runtime Live E2E run into its own subdirectory
(instead of flat-copying, which silently collapsed multiple runs' same-named
records into one observation) and to widen discovery to every successful run
since the previous release tag of any kind.
Adds `spacedock-release journey-delta`, which selects the latest-by-captured_at
observation per scenario/model from the previously published release ledger,
diffs it against the current PR run's freshly emitted records, and posts the
result as a single sticky-updating PR comment (gh pr comment --edit-last).

Wires a new journey-delta-comment job into runtime-live-e2e.yml, gated to
pull_request events with its own pull-requests: write permission.
Adds two spacedock-release CLI primitives the AC-4 backfill runbook composes:
shallow-boot-window-record extracts a shallow-boot-window observation from an
archived claude-stream.jsonl (reusing AC-1's extraction logic, hoisted out of
its test file into internal/ensigncycle so a non-test binary can import it),
and ledger-diff proves a rebuilt journey-costs ledger equals the backed-up
original plus exactly the named added scenarios before any `gh release upload
--clobber` runs.

Both subcommands get CLI-level tests (arg parsing, file I/O, error paths)
matching this file's existing per-subcommand test convention, on top of the
already-covered package-level logic (BuildShallowBootWindowRecord,
DiffAddedScenarios). Fixes a dangling main.go doc-comment reference to a
scripts/backfill-shallow-boot-window.sh that was never created.

Does not run the live gh release upload or the actual 3-release backfill —
that stays an explicit captain-flagged manual step.
Validation's detached adversarial audit found that reverting the "Download
latest journey metrics artifacts" step's per-run cp target back to the flat-
copy overwrite bug (the exact collision AC-2 fixes) left the full test suite
green — the Go-side consumer (AggregateLedger/ReadRecordsDir) was fixture-
tested, but the shell script producing the layout never was.

Extends internal/release/journey_workflow_test.go's existing extractStepRun +
stubbed-gh harness with a positive-path test (two stubbed runs, same record
filename, different content, asserts both land in their own subdirectory
intact) and an adversarial twin that reverts the cp target and asserts the
two runs collapse into one file — proving the positive test actually
discriminates a fixed script from a broken one.
clkao pushed a commit that referenced this pull request Jul 3, 2026
clkao pushed a commit that referenced this pull request Jul 3, 2026
Spike Test added 3 commits July 3, 2026 10:50
…t Minors

C1 (production bug): runtime-live-e2e.yml's journey-delta-comment job hardcoded
--metrics-dir to a path that doesn't match download-artifact's real nested
layout (verified against run 28432388663's zip), REDing the job on every PR.
Adds a "Locate this run's journey metrics" step using release.yml's own
layout-agnostic find+cp pattern, tested against a realistic nested fixture.

C2: BuildShallowBootWindowRecord's BaselineTokens (turns[0].Usage) does not
reconstruct the removed teamRecacheSpikeFloor signal for a multi-turn pre-greet
window, since the spike can land on any pre-greet turn, not just the first.
Adds PreGreetPeakCacheCreation (max cache_creation across pre-greet turns) —
the exact aggregate the original threshold check compared — with a regression
test proving BaselineTokens alone misses a spike not on turn 0.

C4: corrects BaselineTokens' documented semantics — per-turn usage is not
cumulative, so only Tokens.Context() - BaselineTokens.Context() is a
defensible derived quantity, not a full-struct subtraction.

TRIM (captain-ordered de-scope): removes the AC-4 backfill's shipped CLI
subcommands (shallow-boot-window-record, ledger-diff) and internal/release's
ledgerdiff package + tests (~530 lines) in favor of a documented runbook using
jq -S .scenarios | diff plus a throwaway go run script over the exported
BuildShallowBootWindowRecord — the extraction/safeguard logic stays reachable,
just not shipped as permanent CLI surface for a one-time procedure. Also
removes three named meta-tests that verified test-fixture setup rather than
product behavior.

Minors: the PR delta comment now shows cache_read/cache_creation deltas
separately (not just the total — they differ ~12x in cost and meaning for a
boot metric); a no-baseline row renders "n/a (new)" instead of a self-delta
against an implicit zero baseline; the sticky comment is now found by its
marker and updated by id (gh api PATCH) instead of --edit-last, which could
target the wrong comment if another automated comment interleaves.
… steps

Validation's cycle-3 adversarial audit found that reverting ONLY the "Post
the journey-cost delta PR comment" step's --metrics-dir back to the old
broken hardcoded path left the full suite green — nothing verified the Post
step stayed wired to the "Locate this run's journey metrics" step's actual
output directory. Same class of gap Cycle 1 found: a step's own mechanism
tested in isolation, the wiring between two steps not.

Adds TestJourneyDeltaLocateAndPostStepsShareMetricsDir, which extracts both
steps' real run: blocks and derives each step's directory value from the
actual script text (not a hardcoded assumption), then asserts they match.
Verified it catches the exact adversarial mutation validation performed.

Also folds in the two minors from the same review: fixes a stale "backfill
CLI" doc-comment reference in shallow_boot_window_record.go left over from
the AC-4 TRIM (should read "runbook"), and adds
TestRenderJourneyDeltaCommentShowsTokenClassBreakdown, which checks the PR
comment's Cache Read Δ / Cache Creation Δ table cells land in the correct
(non-swapped) columns — verified it catches a reversion to the old
Turns/Tokens-total/Cost table shape.
…-metrics-record-and-pr-delta

# Conflicts:
#	cmd/spacedock-release/main.go
clkao pushed a commit that referenced this pull request Jul 3, 2026
Extend JourneyDelta with DurationMSDelta, computed in ComputeJourneyDeltas
from the existing journeymetrics.Record.DurationMS field, and render it as
a new Duration Δ (ms) column in RenderJourneyDeltaComment.
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Journey cost delta

Scenario Runtime Model Turns Δ Cache Read Δ Cache Creation Δ Tokens Δ (total) Cost Δ (USD) Duration Δ (ms) Baseline
feedback-3-cycle-escalation claude claude-opus-4-8 -2 -86973 -187 -85734 -0.0289 +2823 latest published
feedback-3-cycle-escalation claude sonnet +0 +148764 +6204 +160559 +0.0952 +12665 latest published
filing claude claude-opus-4-8 +0 -1254 -71 -134 +0.0087 +2994 latest published
filing claude sonnet +4 +252528 +7288 +265486 +0.1331 +15267 latest published
gate-guardrail claude claude-opus-4-8 +0 -11583 -1177 -10708 +0.0181 +17285 latest published
gate-guardrail claude sonnet -1 +70008 +3216 +76990 +0.0346 -15848 latest published
merge-hook-guardrail claude claude-opus-4-8 +1 +19588 -1973 +18591 +0.0018 -3195 latest published
merge-hook-guardrail claude sonnet +1 +91695 +2416 +98370 +0.0455 +5430 latest published
rejection-flow claude claude-opus-4-8 -6 -318643 -29851 -353369 -0.5076 -74449 latest published
rejection-flow claude sonnet +26 -986501 -38014 -1034383 +1.1538 +103209 latest published
shallow-boot claude claude-opus-4-8 -5 -235022 -3775 -240144 -0.1920 -33902 latest published
shallow-boot claude sonnet +1 +196165 +4212 +203516 +0.0651 -43512 latest published
shallow-boot-window claude claude-opus-4-8 -5 -3474 -61 -3542 +0.0000 +0 latest published
shallow-boot-window claude sonnet +1 +10718 -792 +9923 +0.0000 +0 latest published

clkao pushed a commit that referenced this pull request Jul 3, 2026
@clkao clkao merged commit faa2846 into main Jul 3, 2026
7 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant